Method and apparatus for classifying signals, method and apparatus for generating descriptors and method and apparatus for retrieving signals

ABSTRACT

The input signal can be quickly and accurately classified and a descriptor can be generated according to the result of classification. Then, the input signal can be retrieved on the basis of the result of classification or the descriptor. A signal processing apparatus comprises a time block splitting section  3  for splitting an audio signal into blocks that are typically  1  second long, a feature extracting section 4 for extracting a characteristic quantity of 18 degrees on the signal attribute from the audio signal in each block and a vector quantizing section  5  for carrying out an operation of categorical classification for the audio signal of each block by means of a vector quantization technique that uses a VQ code book  8  and a characteristic vector formed from the characteristic quantity of 18 degrees. The vector quantizing section  5  outputs a classification label obtained as a result of the categorical classification and a descriptor indicating the reliability of the label. If a signal retrieving operation is conducted in the downstream, the result of the classification or the descriptor is used for the signal retrieval.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to a method and an apparatus for efficientlyclassifying pieces of multimedia information such as video signals andaudio signals, to a method and an apparatus for generating descriptors(tags) corresponding to the classification and also to a method and anapparatus for retrieving input signals according to the result of theclassification or the generated descriptors.

2. Related Background Art

It has been widely recognized that, in order to handle multimediainformation such as video signals and audio signals, it is necessary toclassify video signals and audio signals according to their contents andput an attribute information (tag) to each signal according to thecontents of the signal.

Now, known techniques of classifying signals according to the contentswill be briefly discussed in term of audio signals that are popularlyused for multimedia information.

Generally, an audio signal comprises sounded spans where sounds existand soundless spans where no sound exists. Thus, many known techniquesadapted to classify the attributes of audio signals that can incessantlychange are designed to detect the soundless spans of audio signals. Thesignal whose soundless spans are detected is tagged to show itssoundless spans. Then, the subsequent signal processing operation willbe so controlled that the operation is suspended for the soundless spansindicated by the tag.

Meanwhile, Japanese Patent Application Laid-Open No. 10-207491 disclosesan audio signal classifying technique that consists in classifyingsounds into background sounds and front sounds. With the technique asdisclosed in the above patent document, the power and the spectrum ofthe background sound is estimated and compared with the power and thespectrum of the input signal to isolate background sound spans fromfront sound spans.

While the technique as disclosed in the above patent document iseffective when the input signal is a voice signal and the backgroundsound is a relatively constant and sustained sound, it can no longercorrectly classify input signals if they includes ordinary audio signalssuch as those of music and acoustic signals.

Japanese Patent Application Laid-Open No. 10-187128 discloses atechnique of video signal classifying technique of determining the typeof picture of the input signals that include auxiliary audio signalssuch as voice signals, music signals and/or acoustic signals on thebasis of the sound information accompanying the video information. Thus,with this technique, it is possible to classify audio signals such asvoice signals, music signals and acoustic signals. According to thedisclosed technique, firstly signals showing a predetermined spectrumstructure are classified as music signals and removed from the inputsignals. Then signals showing another spectrum structure are classifiedas voice signals and removed from the remaining signals. Subsequently,signals showing still another spectrum structure are classified asacoustic signals and removed from the remaining signals.

However, since the technique disclosed in the above patent documentregards only spans where the line spectrum structure constantlycontinues as music signals, it cannot reliably be applied to musicsignals that contains signals for sounds of percussion instruments andthose of a song. Additionally, since voice spans are determined on thebasis of the residue left as a result of removing stable line spectrumcomponents (music components) from the original spectrum of the inputsignals, voice signals and acoustic signals cannot be accurately andreliably discriminated from each other.

BRIEF SUMMARY OF THE INVENTION

In view of the above described circumstances, it is therefore the objectof the present invention to provide a method and an apparatus forefficiently and accurately classifying pieces of multimedia informationsuch as video signals and audio signals, a method and an apparatus forgenerating tags (descriptors) corresponding to the classification andalso a method and an apparatus for retrieving input signals according tothe result of the classification or the generated descriptors so thatinput signals may be processed quickly and accurately.

According to the invention, the above object is achieved by providing amethod for classifying signals comprising:

dividing an input signal into blocks having a predetermined time length;

extracting one or more than one characteristic quantities of a signalattribute from the signal of each block; and

classifying the signal of each block into a category according to thecharacteristic quantities thereof.

In another aspect of the invention, there is provided an apparatus forclassifying signals comprising:

a blocking means for dividing an input signal into blocks having apredetermined time length;

a feature extracting means for extracting one or more than onecharacteristic quantities of a signal attribute from the signal of eachblock; and

a categorical classifying means for classifying the signal of each blockinto a category according to the characteristic quantities thereof.

In still another aspect of the invention, there is provided a method forgenerating descriptors comprising:

dividing an input signal into blocks having a predetermined time length;

extracting one or more than one characteristic quantities of a signalattribute from the signal of each block;

classifying the signal of each block into a category according to thecharacteristic quantities thereof; and

generating a descriptor for the signal according to the category ofclassification thereof.

In a further aspect of the invention, there is provided an apparatus forgenerating descriptors comprising:

a blocking means for dividing an input signal into blocks having apredetermined time length;

a feature extracting means for extracting one or more than onecharacteristic quantities of a signal attribute from the signal of eachblock;

a categorical classifying means for classifying the signal of each blockinto a category according to the characteristic quantities thereof; and

a descriptor generating means for generating a descriptor for the signalaccording to the category of classification thereof.

In a still further aspect of the invention, there is provided method forretrieving input signals comprising:

dividing an input signal into blocks having a predetermined time length;

extracting one or more than one characteristic quantities of a signalattribute from the signal of each block;

classifying the signal of each block into a category according to thecharacteristic quantities thereof; and

retrieving the signal according to the result of categoricalclassification or by using a descriptor generated according to theresult of categorical classification.

In still further aspect of the invention, there is provided an apparatusfor retrieving input signals comprising:

a blocking means for dividing an input signal into blocks having apredetermined time length;

a feature extracting means for extracting one or more than onecharacteristic quantities of a signal attribute from the signal of eachblock;

a categorical classifying means for classifying the signal of each blockinto a category according to the characteristic quantities thereof; and

a signal retrieving means for retrieving the signal according to theresult of categorical classification or by using a descriptor generatedaccording to the result of categorical classification.

Thus, according to the invention, a signal that is input continuouslyfor a long period of time is divided into blocks having a predeterminedtime length and the characteristic quantity of a signal attribute isextracted from the signal of each block so that the signal of the blockis automatically classified into a category both according to thecharacteristic quantity. According to the invention, signals areclassified according to the sound sources including voice, music andenvironmental sound and also according to the sound structures in termsof the sounds found in the block, the way how they overlap each otherand the way how they are linked each other without relying on the soundsources of individual sounds such as silence, sounds of single soundsources, those of double sound sources and changing sound sources. Thus,according to the invention, audio signals are classified both accordingto the sound sources and according to the sound structures to make itpossible to reliably and efficiently classify various acoustic scenesthat occur successively. Note that the predetermined time length of eachblock is such a one with which the signal attribute in the block can beclearly identified and the signal structure of the block can beclassified in a simple fashion. Preferably, it may be a second, althoughthe predetermined time length of each bock according to the invention isby no means limited to a second and may alternatively have any otherappropriate value. Still alternatively, the time length of the blockdoes not necessarily have to have a single value and may be variablefrom block to block. More specifically, several time lengths may beselectively used or the time length of the block may be made adaptivelyvariable without departing from the scope of the present invention.

As described above, with a method and an apparatus for classifyingsignals according to the invention, it is possible to classify inputsignals quickly and accurately by dividing an input signal into blockshaving a predetermined time length, extracting the characteristicquantity of a signal attribute from the signal of each block andclassifying the signal of each block into a category according to thecharacteristic quantity thereof. Therefore, it is now possible toclassify the type of the sound source and that of the structure of eachof the blocks of an audio signal that is a time series signal wherevarious sound sources show various different patterns over a long periodof time.

With a method and an apparatus for generating descriptors according tothe invention, it is now possible to automatically select an appropriaterecognition method and a coding method for any given audio signal bygenerating a descriptor for the signal according to the category ofclassification thereof because a specific sound span of the audio signalcan be identified and used for a preprocessing operation to be conductedfor the purpose of voice recognition or acoustic signal coding to nameonly a few.

With a method and an apparatus for retrieving input signals according tothe invention, for example, the point of switch of sound sources and theclassifications of the sound source of an input signal can be retrievedby retrieving the signal so that it is now possible to automaticallydetect the point of switch of topics or that of television programs andhence multimedia data can be retrieved with ease. Additionally, with amethod and an apparatus for retrieving input signals according to theinvention, it is now possible to improve the accuracy of detecting ascene change by viewing pictures with a cut change detection feature.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

FIG. 1 is a schematic block diagram of a first embodiment of theinvention, which is a signal processing an apparatus, schematicallyillustrating its configuration;

FIG. 2 is a schematic illustration of an operation of blocking an audiosignal;

FIG. 3 is a schematic block diagram of the feature extracting means ofFIG. 1, illustrating a specific configuration thereof;

FIG. 4 is a schematic illustration of structural classificationcategories;

FIG. 5 is a flow chart of the processing operation conducted on eachblock by the vector extracting means of FIG. 1;

FIG. 6 is a schematic block diagram of the function of the vectorquantizing means of FIG. 1 to be used when classifying the audio signalof a block as a changing sound or a non-changing sound;

FIG. 7 is a schematic block diagram of the function of the vectorquantizing means of FIG. 1 to be used when classifying the audio signalof a block as voice, music, environmental sound and so on;

FIG. 8 is a schematic block diagram of a second embodiment of theinvention, which is a signal processing an apparatus, schematicallyillustrating its configuration; and

FIG. 9 is a flow chart of the processing operation conducted by thesecond embodiment when retrieving a desired audio signal by detecting ascene change of the audio signal.

DETAILED DESCRIPTION OF THE INVENTION

Now, the present invention will be described by referring to theaccompanying drawings that illustrate preferred embodiments of theinvention.

FIG. 1 is a schematic block diagram of a first embodiment of the presentinvention, which is a signal processing an apparatus adapted to classifyinput signals (e.g., audio signals), schematically illustrating itsconfiguration.

Referring to FIG. 1, an audio signal is input to input terminal 1 andtemporarily stored in buffer memory 2. Subsequently, it is read out andsent to time block splitting section 3.

The time block splitting section 3 divides the audio signal fed from thebuffer memory 2 into blocks having a predetermined time length (timeblock division) and sends the obtained blocks of audio signal to featureextracting section 4. The blocking operation of the time block splittingsection 3 will be described in greater detail hereinafter.

The feature extracting section 4 extracts a plurality of characteristicquantities from each block of audio signal and send them to vectorquantizing section 5. The processing operation of the feature extractingsection 4 for extracting characteristic quantities will be described ingreater detail hereinafter.

The vector quantizing section 5 uses a so-called vector quantitytechnique as will be described in greater hereinafter. It compares thevector (to be referred to as characteristic vector hereinafter) formedby the plurality of characteristic quantities fed from the featureextracting section 4 with a VQ code book (vector quantization code book)8 containing a set of a plurality of centroids (centers of gravity in apattern space) generated in advance by learning, searches for thecentroid showing a Mahalanobis distance that is closest to saidcharacteristic vector and outputs the representative codes representedby said closest centroid. More specifically, with this embodiment, therepresentative codes output from the vector quantizing section 5 are theclassification label that corresponds to the sound source classificationcategory and the classification label that corresponds to the structuralclassification category of the audio signal. In other words, the vectorquantizing section 5 outputs the result of the operation of classifyingthe audio signal according to the characteristic vector. In the case ofthis embodiment, the vector quantizing section 5 is adapted to outputthe reciprocal of the shortest distance to the above searched centroidas index showing the reliability of the classification of the categoryalong with the above classification label. Then, in this embodiment ofthe invention, the classification label obtained by the structuralclassification and its reliability as well as the classification labelobtained by the sour source classification and its reliability outputfrom the vector quantizing section 5 are outputs from terminal 6 assignal descriptors representing the result of classification. Referringto FIG. 1, the operation of the time block splitting section 3 forsplitting the input audio signal into blocks, that of the featureextracting section 4 for extracting characteristic quantities of theaudio signal of each block and that of the vector quantizing section 5for classifying the audio signal of each block will be described indetail.

Firstly, the operation of the time block splitting section 3 of FIG. 1for splitting the input audio signal into blocks (time block division)will be discussed.

The time block splitting section 3 is adapted to split an audio signalthat is given as various time series sounds and extends over a longperiod of time into time blocks having an appropriate time length inorder to facilitate the subsequent classifying operation.

It will be appreciated that an operation of classifying an audio signalthat last for tens of several seconds into a category is impractical andnot feasible because the signal can include sounds of various differenttypes and various different sound patterns. On the other hand, thesignal pattern that changes with time is essential when classifyingsounds and hence it is not feasible to divide an audio signal intosignal elements that last only tens of several milliseconds anddetermines the category to which each signal element belongs, ifcategories are established in terms of voice/music/noise.

Thus, in this embodiment, the time block splitting section 3 is adaptedto split an audio signal into blocks having a time length of 1 second inorder to meet the requirements that “the attribute of each signalelement produced by splitting an audio signal can be accuratelyidentified” and that “the structure of each signal element produced bysplitting an audio signal can be classified in a simple manner”.

Additionally, in this embodiment, each block is made to overlap anadjacent block by a time length that is equal to a half of that of theblock as shown in FIG. 2 in order to enhance the accuracy of thesubsequent classifying operation. More specifically, the time blocksplitting section 3 of this embodiment produces blocks B0, B1, B2, B3, .. . having a time length of 1 second and makes the latter half of blockB0 overlap the former half of block B1, the latter half of block B1overlap the former half of block B2, the latter half of block B2 overlapthe former half of block B3, the latter half of block B3 overlap theformer half of block B4 and so on.

Now, the operation of the feature extracting section 4 of FIG. 1 forextracting characteristic quantities of the signal of each block(feature extraction) will be discussed below.

The feature extracting section 4 is adapted to extract characteristicquantities suitable for the subsequent classifying operation from thesignal of each block produced by the time block splitting section 3.

Now, the characteristic quantities of each block extracted by thefeature extracting section 4 will be discussed in detail. In thefollowing description, t stands for a variable representing time, Tstands for the length of each block (=1 second) and i stands for theblock number while s_(i) (t) stands for the signal of the first block(0≦t≦T), ω stands for a variable representing frequency and Ω stands forthe upper limit of frequency (which is equal to a half of the samplingfrequency when the processing operation of the present invention isrealized discretely). Furthermore, S_(i) (t, w) stands for thespectrogram of the signal of the first block (0≦t≦T, 0≦ω≦Ω) and E[ ]stands for the average time period of a number of blocks while V[ ]stands for the temporal relative standard deviation of a number ofblocks (the value obtained by standardizing the square root of varianceswith the average).

The feature extracting section 4 computationally determines a total ofeighteen (18) characteristic quantities of the signal of each blockincluding the average P_(m) and the standard deviation P_(sd) of thesignal power in the block, the average W_(m) and the standard deviationW_(sd) of the spread of the spectrogram of the signal in the block, theaverage L_(m) and the standard deviation L_(sd) of the power of the lowfrequency component of the signal in the block, the average M_(m) andthe standard deviation M_(sd) of the power of the intermediate frequencycomponent of the signal in the block, the average H_(m) and the standarddeviation H_(sd) of the power of the high frequency component of thesignal in the block, the average F_(m) and the standard deviation F_(sd)of the pitch frequency of the signal in the block, the average A_(m) andthe standard deviation A_(sd) of the degree of harmonic structurizationof the signal in the block, the average R_(m) and the standard deviationR_(sd) of the LPC (linear predictive analysis) residual energy of thesignal in the block and the average G_(m) and the standard deviationG_(sd) of the pitch gain of the LPC residual signal of the signal in theblock.

The average P_(m) and the standard deviation P_(sd) of the signal powerin the block are expressed respectively by formula (1) and (2) below.P _(m) =E[s ²(t)]  (1)P _(sd) =V[s ²(t)]  (2)

The average W_(m) and the standard deviation W_(sd) of the spread of thespectrogram of the signal in the block are expressed respectively byformula (3) and (4) below. Note that, in this embodiment, a total offive hundreds and twelve (512) samples obtained (by every 31.25milliseconds) by using a sampling frequency of 16 kHz are used for thespectrum:W _(m) =E[w(t)]  (3) andW _(sd) =V[w(t)]  (4)where w (t) is expressed by formula (5) below and represents a frequencywidth where the spectrogram exceeds a given threshold value.Particularly, the frequency width expressed by the above formula w (t)tends to be wide and constant in the case of music, whereas it does notremain constant and tends to widely vary in the case of voice.Therefore, the frequency width of w (t) can be used as a characteristicquantity of music, voice and other sounds. $\begin{matrix}{{{w(t)} = {\frac{1}{\Omega}{\int_{\Gamma}^{\quad}\quad{\mathbb{d}\omega}}}},{\Gamma = \left\{ {\omega\left. {{{Si}\left( {t,\omega} \right)} > {Threshold}} \right\}} \right.}} & (5)\end{matrix}$

The average L_(m) and the standard deviation L_(sd) of the power of thelow frequency component of the signal in the block are expressedrespectively by formula (6) and (7) below. Note that, in thisembodiment, a frequency band between 0 and 70 Hz is used for the lowfrequency component:L _(m) =E[l(t)]  (6) andL _(sd) =V[l(t)]  (7)where l (t) is expressed by formula (8) below and represents thestandardized power of the low frequency component of the signal at timet. Particularly, voice practically does not contains any component of 70Hz and below, whereas the sounds of percussion instruments such as drumsnormally contain a number of frequency components of 70 Hz and below.Therefore, the low frequency component can be used as a characteristicquantity of music, voice and other sounds. $\begin{matrix}{{{1(t)} = \frac{\int_{\omega\quad 0}^{\omega\quad 1}{{S_{i}\left( {t,\omega} \right)}\quad{\mathbb{d}\omega}}}{\int_{0}^{\Omega}{{S_{i}\left( {t,\omega} \right)}\quad{\mathbb{d}\omega}}}},\left( {{\omega_{0} = {0\quad{Hz}}},{\omega_{1} = {70\quad{Hz}}},} \right)} & (8)\end{matrix}$

The average M_(m) and the standard deviation M_(sd) of the power of theintermediate frequency component of the signal in the block areexpressed respectively by formula (9) and (10) below. Note that, in thisembodiment, a frequency band between 70 Hz and 4 kHz is used for theintermediate frequency component:M _(m) =E[m(t)]  (9) andM _(sd) =V[m(t)]  (10)where m (t) is expressed by formula (11) below and represents thestandardized power of the intermediate frequency component of the signalat time t. Particularly, voice is almost totally contained in thefrequency band between 70 Hz and 4 kHz. Therefore, the intermediatefrequency component can be used as a characteristic quantity of music,voice and other sounds. $\begin{matrix}{{{m(t)} = \frac{\int_{\omega\quad 1}^{\omega\quad 2}{{S_{i}\left( {t,\omega} \right)}\quad{\mathbb{d}\omega}}}{\int_{0}^{\Omega}{{S_{i}\left( {t,\omega} \right)}\quad{\mathbb{d}\omega}}}},\left( {{\omega_{1} = {70\quad{Hz}}},{\omega_{2} = {4\quad{kHz}}},} \right)} & (11)\end{matrix}$

The average H_(m) and the standard deviation H_(sd) of the power of thehigh frequency component of the signal in the block are expressedrespectively by formula (12) and (13) below. Note that, in thisembodiment, a frequency band between 4 kHz and 8 kHz is used for thehigh frequency component:H _(m) =E[h(t)]  (12) andH _(sd) =V[h(t)]  (13)where h (t) is expressed by formula (14) below and represents thestandardized power of the high frequency component of the signal at timet. Particularly, voice practically does not contains any component of 4kHz and above, whereas the sounds of percussion instruments such ascymbals normally contain a number of frequency components between 4 kHzand 8 kHz. Therefore, the high frequency component can be used as acharacteristic quantity of music, voice and other sounds.$\begin{matrix}{{{h(t)} = \frac{\int_{\omega\quad 2}^{\omega\quad 3}{{S_{i}\left( {t,\omega} \right)}\quad{\mathbb{d}\omega}}}{\int_{0}^{\Omega}{{S_{i}\left( {t,\omega} \right)}\quad{\mathbb{d}\omega}}}},\left( {{\omega_{2} = {4\quad{kHz}}},{\omega_{3} = {8\quad{kHz}}},} \right)} & (14)\end{matrix}$

The average F_(m) and the standard deviation F_(sd) of the pitchfrequency of the signal in the block are expressed respectively byformula (15) and (16) below:F _(m) =E[f(t)]  (15) andF _(sd) =V[f(t)]  (16)where f (t) represents the pitch frequency of the signal at time t,which is typically determined by using Parson's technique (T. Parson:Separation of Speech from Interfering Speech by ms of HarmonicSelection; J. Acoust. Soc. Am., 60,4, 911/918 (1976). Particularly, thepitch frequency is used for extracting the characteristic quantity ofthe degree of harmonic structurization as will be described hereinafterand generally differs between music and voice and between male voice andfemale voice so that it can be used as a characteristic quantity of suchsounds.

The average A_(m) and the standard deviation A_(sd) of the degree ofharmonic structurization of the signal in the block (which is expressedby a (t) in this embodiment) are expressed respectively by formula (17)and (18) below:A _(m) =E[a(t)]  (17) andA _(sd) =V[a(t)]  (18)where a (t) is expressed by formula (19) below and represents the ratioof the energy of the sound component of integer times of the pitchfrequency to the energy of all the frequencies. Additionally, Arepresents a micro frequency such as ±15 Hz. Particularly, the degree ofharmonic structurization is remarkably reduced for noise-like sounds.Therefore, the degree of harmonic structurization can be used as acharacteristic quantity of noise-like sounds and other sounds.$\begin{matrix}\begin{matrix}{{{a(t)} = \frac{\int_{\Gamma}^{\quad}{{S_{i}\left( {t,\omega} \right)}\quad{\mathbb{d}\omega}}}{\int{{S_{i}\left( {t,\omega} \right)}{\mathbb{d}\omega}}}},\Gamma} \\{= \left\{ {\omega\quad\left. {{{{{nf}(t)} - \Delta} \leq \omega \leq {{{nf}(t)} + \Delta}},{n = 1},2,\ldots} \right\}} \right.}\end{matrix} & (19)\end{matrix}$

The average R_(m) and the standard deviation R_(sd) of the LPC (linearpredictive analysis) residual energy of the signal in the block areexpressed respectively by formula (20) and (21) below:R _(m) =E[r ²(t)]/E[s ²(t)]  (20) andR _(sd) =V[r ²(t)]/V[s ²(t)]  (21)where r (t) represents the residue signal of the LPC analysis (which istypically conducted on the basis of 30 mn frame and 12 degrees). Theyare quantities of evaluating the complexity of the spectrum structure inthe block (in terms of noises and consonants) and determinedrespectively as ratios relative to average and the standard deviation ofthe power of the original signal. Therefore the LPC residual energy canbe used as characteristic quantity of noises, consonants and othersounds.

The average G_(m) and the standard deviation G_(sd) of the pitch gain ofthe LPC residual signal of the signal in the block are expressedrespectively by formula (22) and (23) below:G _(m) =E[g(t)]  (22) andG _(sd) =V[g(t)]  (23)where g (t) represents the maximal value of the short termauto-correlation function at and near time t of r (t) and hence is aquantity of evaluating the degree of periodicity of the residue signalof the LPC analysis (which is typically conducted on the basis of 30 msframe and 12 degrees) in the block. Particularly, the pitch gain of theLPC residue signal shows a remarkably low value for white noises andconsonants, whereas it shows a high value for voice and music.Therefore, the pitch gain of the LPC residual signal can be used ascharacteristic quantity of noises, consonants, voice, music and othersounds.

In this embodiment, a vector as expressed by formula (24) below isformed by using the above described eighteen characteristic quantitiesand used as characteristic vector X_(i) of the block (time block).X _(i) =[P _(m) , P _(sd) , W _(m) , W _(d) , . . . , G _(m) , G_(sd)]  (24)

FIG. 3 is a schematic block diagram of the feature extracting section 4of FIG. 1 for determining the above described characteristic vector of18 degrees, illustrating a specific configuration thereof.

Referring to FIG. 3, the audio signal s_(i) (t) of the i-th blockproduced by the time block division of the time block splitting sectionof FIG. 1 is input to terminal 10. The audio signal s_(i) (t) of thei-th block is then sent to waveform analysing section 11, spectrumanalysing section 12 and LPC analysing section 13.

The waveform analysing section 11 determines the average P_(m) and thestandard deviation P_(sd) of the signal power as described above byreferring for formulas (1) and (2) for the audio signal Si (t) of thei-th bock. Then, the average P_(m) and the standard deviation P_(sd) ofthe signal power are sent to the downstream vector quantizing section 5respectively by way of corresponding terminals 22, 23 as two of thecharacteristic quantities of the vector of eighteen degrees X_(i).

The spectrum analysing section 12 performs a spectrum analysis operationon the audio signal s_(i) (t) of the i-th block and generatesspectrogram S_(i) (t,w) of the signal of the i-th block. The spectrogramS_(i) (t,w) of the signal of the i-th block is then sent to thresholdprocessing section 14, low frequency component extracting section 15,intermediate frequency component extracting section 16, high frequencycomponent extracting section 17, pitch extracting section 18 and degreeof harmonic structurization extracting section 19.

The threshold processing section 14 determines the average W_(m) and thestandard deviation W_(sd) of the spread of the spectrogram as describedabove by referring to formulas (3) and (4), using the spectrogram S_(i)(t,w) of the signal of the i-th block. Then, the average W_(m) and thestandard deviation W_(sd) of the spread of the spectrogram are sent tothe downstream vector quantizing section 5 respectively by way ofcorresponding terminals 24, 25 as two of the characteristic quantitiesof the vector of eighteen degrees X_(i). The low frequency componentextracting section 15 determines the average L_(m) and the standarddeviation L_(sd) of the power of the low frequency component asdescribed above by referring to formulas (6) and (7), using thespectrogram S_(i) (t,w) of the signal of the i-th block. Then, theaverage L_(m) and the standard deviation L_(sd) of the power of the lowfrequency component are sent to the downstream vector quantizing section5 respectively by way of corresponding terminals 26, 27 as two of thecharacteristic quantities of the vector of eighteen degrees X_(i).

The intermediate frequency component extracting section 16 determinesthe average M_(m) and the standard deviation M_(sd) of the power of theintermediate frequency component as described above by referring toformulas (9) and (10), using the spectrogram S_(i) (t,w) of the signalof the i-th block. Then, the average M_(m) and the standard deviationM_(sd) of the power of the intermediate frequency component are sent tothe downstream vector quantizing section 5 respectively by way ofcorresponding terminals 28, 29 as two of the characteristic quantitiesof the vector of eighteen degrees X_(i).

The high frequency component extracting section 17 determines theaverage H_(m) and the standard deviation H_(sd) of the power of the highfrequency component as described above by referring to formulas (12) and(13), using the spectrogram S_(i) (t,w) of the signal of the i-th block.Then, the average H_(m) and the standard deviation H_(sd) of the powerof the high frequency component are sent to the downstream vectorquantizing section 5 respectively by way of corresponding terminals 30,31 as two of the characteristic quantities of the vector of eighteendegrees X_(i).

The pitch extracting section 18 determines the average F_(m) and thestandard deviation F_(sd) of the pitch frequency as described above byreferring to formulas (15) and (16), using the spectrogram S_(i) (t,w)of the signal of the i-th block. Then, the average F_(m) and thestandard deviation F_(sd) of the pitch frequency are sent to thedownstream vector quantizing section 5 respectively by way ofcorresponding terminals 32, 33 as two of the characteristic quantitiesof the vector of eighteen degrees X_(i).

The degree of harmonic structurization extracting section 19 determinesthe average A_(m) and the standard deviation A_(sd) of the degree ofharmonic structurization as described above by referring to formulas(18) and (19), using the spectrogram S_(i) (t,w) of the signal of thei-th block. Then, the average A_(m) and the standard deviation A_(sd) ofthe degree of harmonic structurization are sent to the downstream vectorquantizing section 5 respectively by way of corresponding terminals 34,35 as two of the characteristic quantities of the vector of eighteendegrees X_(i).

The LPC analysing section 13 performs an operation of LPC analysis onthe audio signal S_(i) (t) of the i-th block and generates residuesignal r (t) of the LPC analysis of the i-th block. The generatedresidue signal r (t) of the LPC analysis is sent to residual energyextracting section 20 and pitch gain extracting section 21.

The residual energy extracting section 20 determines the average R_(m)and the standard deviation R_(sd) of the residual energy of the LPCanalysis as described above by referring to formulas (20) and (21),using the residue signal r (t) of the LPC analysis of the i-th block.Then, the average R_(m) and the standard deviation R_(sd) of theresidual energy of the LPC analysis are sent to the downstream vectorquantizing section 5 respectively by way of corresponding terminals 36,37 as two of the characteristic quantities of the vector of eighteendegrees X_(i).

The pitch gain extracting section 21 determines the average G_(m) andthe standard deviation G_(sd) of the pitch gain of the LPC analysis asdescribed above by referring to formulas (22) and (23), using theresidue signal r (t) of the LPC analysis of the i-th block. Then, theaverage G_(m) and the standard deviation G_(sd) of the residual energyof the LPC analysis are sent to the downstream vector quantizing section5 respectively by way of corresponding terminals 38, 39 as two of thecharacteristic quantities of the vector of eighteen degrees X_(i).

Upon receiving the vector of 18 degrees of each block, the vectorquantizing section 5 classifies the audio signal of the block on thebasis of the vector of 18 degrees, using a vector quantizationtechnique. Now, the classes used for classifying the audio signal ofeach block will be detailedly discussed below.

For classifying the audio signal of each block with this embodiment, itis classified into a structural class and a sound source class in amanner as described below.

Firstly the structural classes that are used for the purpose ofclassification of audio signals in this embodiment will be described indetail.

The structural classes refers to an operation of classifying audiosignals not according to the types of sound sources but according to thestructure patterns of the signals in the blocks. In this embodiment, asilence structure (silent), a single sound source structure (single), adouble sound source structure (double), a sound source change structure(change), a multiple sound source change structure (multiple change), asound source partial change structure (partial change) and an extrastructure (other) are defined as structural classification patterns(categories). FIG. 4 is a schematic illustration of structuralclassification categories.

The silence structure pattern refers to a state where no significantsound exists in the block and the block is in a silent state 100.

The single sound source structure pattern refers to a state where only asingle type of significant sound 101 exists substantially over theentire range of the block.

The double sound source structure pattern refers to a state where twotypes of significant sound (sound 102 and sound 103) exist substantiallyover the entire range of the block. It may be a state where voice soundsabove BGM (background music).

The sound source change structure pattern refers to a state where thetype of sound source is switched in the block. For example, voice 104may be switched to music 105. Note that this pattern includes a changefrom significant sound to silence and vice versa.

The multiple sound source change structure pattern refers to a statewhere two sound sources are switched simultaneously in the block (e.g.,two sound sources 106 and 108 may be switched to other two sound sources107, 109). Note that this pattern includes a change from a single soundsource (or silence) to two sound sources (e.g., a single sound source113 may be switched to two sound sources 114, 115) and a change from twosound sources to a single sound source (or silence) (e.g., two soundsources 110 and 111 may be switched to a single sound source 112). Atypical example of this multiple sound source change structure patternmay be a state where both BGM and voice end almost simultaneously.

The sound source partial change structure pattern refers to a statewhere a single type of sound (sound 118) exists substantially over theentire range of the block and a coexisting sound is switched (e.g.,sound 116 is switched to sound 117). Note that his pattern includes achange from sounds of two sound sources to a sound of a single soundsource. A typical example of this sound source partial change structuremay be a state where BGM continues when voice sounding above the BGMsuddenly ends.

The extra structure pattern refers to a state where none of the abovepatterns is applicable. It may be a state where three different sounds(e.g., sounds 119, 120, 121) coexist of a state where more than twoswitches of sound occurs in the block(e.g., sound 122 is switched tosound 123 and then switched further to sound 124).

Now, the sound source classes that are used for the purpose ofclassification of audio signals in this embodiment will be described indetail.

The sound source classes refer to the classification according to thetypes of sound sources. As will be described hereinafter, voice, music,noise, striking sound, environmental sound and other sound are used asfor the classification of sound sources.

Voice refers to human voice and may be further classified intosub-classes of male voice, female voice and other voice (infant voice,artificial voice, etc.).

Music refers to music sound and may be further classified intosub-classes of music sound of instrument, vocal music sound and othermusic sound (e.g., rap music sound).

Noise refers to any white noise that may be generated form machines.

Striking sound refers to the sound of knocking a door, the sound offootsteps, clapping sound (of a limited number of people) and so on thatare generated by striking something. The volume of a striking soundrises abruptly immediately after the generation thereof and thenattenuates. If necessary, striking sound may be further classified intosub-classes according to the sound source.

Environmental sound refers to the sound of hustle and bustle, clappingsound (of a large number of people), cheering sound, engine sound andall other sounds. If necessary, environmental sound may be furtherclassified into sub-classes according to the sound source.

The vector quantizing section 5 of FIG. 1 performs an operation ofclassifying the audio signal of each block into a structural class and asound source class, using the characteristic vector of 18 degrees.

Now, the classifying operation of the vector quantizing section 5 usingthe characteristic vector will be described in detail below.

In this embodiment, the operation of classifying the audio signal ofeach block proceeds in three steps as illustrated in the flow chart ofFIG. 5.

Referring to FIG. 5, upon receiving the characteristic vector X_(i) of18 degrees determined for the i-th block in Step S1, the vectorquantizing section 5 determines in Step S2 if the audio signal of thei-th block is classified into the silence class or not. Morespecifically it determines if it is classified into the silencestructure pattern of the structural classes or not by checking if theaverage P_(m) and the standard deviation P_(sd) of the signal power isbelow a given threshold value or not.

If it is determined in Step S2 that the audio signal of the i-th blockis classified into the silence structure pattern, the vector quantizingsection 5 outputs in Step S6 the result of the operation of classifyingthe audio signal into the silence structure pattern and returns to StepS1 for the operation of processing the audio signal of the next block.On the other hand, it is determined in Step S2 that the audio signal ofthe i-th block is not classified into the silence structure pattern, thevector quantizing section 5 proceeds to the processing operation of StepS3.

In Step S3, the vector quantizing section 5 carries out the processingoperation for change classification. More specifically, the vectorquantizing section 5 determines if the audio signal can be classifiedsignal into any of the sound source change structure (change), themultiple sound source change structure (multiple change) and the a soundsource partial change structure (partial change) or any of the singlesound source structure (single), the double sound source structure(double) and the extra structure (other).

To carry out this classification, the vector quantizing section 5firstly generates a new characteristic vector Y_(i) by using thecharacteristic vector X_(i−1) of the i−1-th block immediately precedingthe i-th block and the characteristic vector X_(i+1) of the i+1-th blockimmediately succeeding the i-th block. In other words, it uses formula(25) below to generate a new characteristic vector Y_(i).Y _(i)=(X _(i+1) −X _(i−1))/(X _(i+1) +X _(i−1))   (25)

Note that this operation of addition, subtraction and division iscarried out for each characteristic quantity of the characteristicvector X_(i−1) and that of the characteristic vector X_(i+1).

After determining the new characteristic vector Y_(i) in a manner asdescribed above, the vector quantizing section 5 compares the newcharacteristic vector Y_(i) and the VQ code book 8 it memorizes inadvance. Then, it retrieves the centroid showing the closest Mahalanobisdistance and finds out the category of the closest centroid (if a changestructure is applicable or not in this case). If it is found in Step 3that a change structure is applicable, the vector quantizing section 5outputs in Step S7 the result of the classifying operation showing thatthe audio signal of the i-th block is classified into the sound sourcechange structure (change), the multiple sound source change structure(multiple change) or the a sound source partial change structure(partial change) along with the reciprocal of the shortest distance tothe centroid (the reliability of the structural classification) obtainedby the above vector quantization. Then, the vector quantizing section 5returns to Step S1 for the operation of processing the audio signal ofthe next block. If, on the other hand, it is determined in Step S3 thatno change structure is applicable, the vector quantizing section 5proceeds to the processing operation of Step S4.

In Step S4, the vector quantizing section 5 carries out an operation ofsource classification of classifying the audio signal into one of thenon-change patterns including the single sound source structure(single), the double sound source structure (double) and the extrastructure (other). Then, in Step S5, it outputs the result of the soundsource classification showing if it is voice, music, noise, strikingsound, environmental sound or other sound. More specifically, the vectorquantizing section 5 employs a vector quantization technique andcompares the characteristic vector X_(i) of 18 degrees of the i-th blockand the VQ code book 8 it memorizes in advance. Then, it retrieves thecentroid showing the closest Mahalanobis distance and outputs theclassification label represented by the closest centroid along with thereciprocal of the shortest distance to the centroid (the reliability ofthe classification of the category) as the result of classification.After the processing operation of Step S5, the vector quantizing section5 returns to Step S1 for the operation of processing the audio signal ofthe next block.

FIG. 6 is a schematic block diagram of the function to be used when theoperation of Step S3 and that of Step S7 of the flow chart of FIG. 5 arecarried out by the vector quantizing section 5 and the VQ code booksection 8 of FIG. 1, whereas FIG. 7 is a schematic block diagram of thefunction to be used when the operation of Step S4 and that of Step S5 ofthe flow chart of FIG. 5 are carried out by the vector quantizingsection 5 and the VQ code book section 8 of FIG. 1. In other words, whencarrying out the operation of Step S3 and that of Step S7 of FIG. 5, thevector quantizing section 5 and the VQ code book section 8 of FIG. 1functionally operate in a manner as illustrated in FIG. 6. On the otherhand, when carrying out the operation of Step S4 and that of Step S5 ofFIG. 5, the vector quantizing section 5 and the VQ code book section 8of FIG. 1 functionally operate in a manner as illustrated in FIG. 7.While the functional operation in the vector quantizing section 5 isillustrated in two drawings of FIGS. 6 and 7 for the purpose of easyunderstanding, the vector quantizing section 5 is by no meansfunctionally divided into two parts. In other words, the vectorquantizing section 5 operates either in a manner as illustrated in FIG.6 or in a manner as illustrated in FIG. 7 depending on the result of theprocessing operation of Step S2 and that of Step S3 of the flow chart ofFIG. 5.

Referring firstly to FIG. 6, the characteristic vector X_(i−1) of thei−1-th block that immediately precedes the i-th block to be classifiedis supplied to terminal 51 of the vector quantizing section 5, while thecharacteristic vector X_(i+1) of the i+1-th block that immediatelysucceeds the i-th block is supplied to terminal 52 of the vectorquantizing section 5. The characteristic vector X_(i−1) of the i−1-thblock and that of the characteristic vector X_(i+1) of the i+1-th blockare then sent to feature mixing arithmetic operation section 53 in thevector quantizing section 5.

The feature mixing arithmetic operation section 53 mixes thecharacteristic vector X_(i−1) of the i−1-th block and the characteristicvector X_(i+1) of the i+1-th block to generate a new characteristicvector Y_(i) by using the formula (25) for mixing features. Thegenerated new characteristic vector Y_(i) is then sent to section 54 forcomputation of distance, arithmetic operation for comparison, which is aprincipal component of the vector quantizing section 5.

The section 54 for computation of distance, arithmetic operation forcomparison compares the new characteristic vector Y_(i) and the VQ codebook 8. Then, it retrieves the centroid showing the Mahalanobis distanceclosest to the characteristic vector Y_(i) and outputs the categoryrepresented by the centroid as the result of classification (changingsound or non-changing sound). The descriptor showing the result ofclassification is output from output terminal 55 of the vectorquantizing section 5.

Referring now to FIG. 7, the characteristic vector X_(i) of the i-thblock to be classified is supplied to terminal 61 of the vectorquantizing section 5. Then, the characteristic vector X_(i) of the i-thblock is sent to section 62 for computation of distance, arithmeticoperation for comparison, which is also a principal component of thevector quantizing section 5.

The section 62 for computation of distance, arithmetic operation forcomparison compares the characteristic vector X_(i) and the VQ code book8. Then, it retrieves the centroid showing the Mahalanobis distanceclosest to the characteristic vector X_(i) and outputs the categoryrepresented by the centroid as the result of classification (voice,music, noise, environmental sound, etc.). The descriptor showing theresult of classification is output from output terminal 63 of the vectorquantizing section 5.

As described above, with the first embodiment of signal processing anapparatus according to the present invention can classify the type ofthe sound source and that of the structure of each of the blocks of anaudio signal that is a time series signal where various sound sourcesshow various different patterns over a long period of time and whichtypically represents various sounds including voices, music,environmental sounds and noises that are emitted simultaneously orcontinuously in an overlapping manner. Additionally, with thisembodiment of signal processing an apparatus, it is now possible toidentify sound segments so that they may be used for a preliminaryprocessing operation for voice recognition and coding of acousticsignals so as to automatically select an appropriate recognition methodand a coding method for any given audio signal.

Now, a second embodiment of the present invention will be describedbelow.

When, for example, retrieving a necessary part of the stream of anaccumulated long audio signal, generally, the user may listen to thestream of sound while replaying it in the fast replay mode and startsreplaying it in the normal replay mode when he or she locates the startof the wanted part. However, with this retrieving technique, it willtake a long time before the user can locate the wanted part of the audiosignal and the user is forced to endure the tedious operation oflistening to the queer sound produced as a result of the fast replay.

With the second embodiment of the present invention, the result ofclassification of sound change structure (particularly, the sound sourcechange structure and the multiple sound source change structure) asdescribed above by referring to the first embodiment is used to detectthe point(s) of switch of the audio signal (to be referred to as scenechange(s) hereinafter) and the normal replay operation is made to startat the time when a scene change from a silence structure to some otherstructure is detected in order to facilitate the retrieval of the audiosignal.

FIG. 8 is a schematic block diagram of a second embodiment of theinvention, which is a signal processing an apparatus, schematicallyillustrating its configuration that is adapted to use the result ofclassification of sound change structure (particularly, the sound sourcechange structure and the multiple sound source change structure)obtained by means of the technique of classifying audio signalsdescribed above by referring to the first embodiment in order tofacilitate the retrieval of a wanted audio signal. FIG. 9 is a flowchart of the processing operation conducted by the second embodimentwhen retrieving a wanted audio signal by detecting a scene change of theaudio signal.

Now, the configuration and the operation of the second embodiment ofsignal processing an apparatus will be described by referring to FIGS. 8and 9.

Referring firstly to FIG. 8, replay section 71 adapted to replay anaudio signal from any of various different information recording mediaand telecommunications media and output it under the control of replaycontrol block 77, which will be described hereinafter. When retrieving adesired audio signal by means of this second embodiment, an audio signalis output from the replay section 71 in a fast replay mode and input toclassifying section 74 that operates like the first embodiment of signalprocessing an apparatus.

Referring to FIG. 9, the classifying section 74 carries out in Step S11a classifying operation on the audio signal that is reproduced in thefast replay mode, using the above described techniques of blocking,characteristic extraction and vector quantization and outputs adescriptor (tag) showing the result of classification of each block. Thedescriptor is then sent to downstream scene change detecting section 75.

Alternatively, it is also possible for the classifying section 74 tocarry out a classifying operation on the audio signal in advance andsynchronously adds the descriptor to the audio signal so that the audiosignal accompanied by the descriptor may be output from the replaysection 71. It will be appreciated, however, that if it is so arrangedthat the replay section 71 outputs the audio signal accompanied by thedescriptor, the classifying operation of the classifying section 74 isskipped and the descriptor is directly input to the scene changedetecting section 75.

Then, upon receiving the descriptor showing the result of classificationof the block from the classifying section 74 in Step S12, the scenechange detecting section 75 checks in Step S13 if the audio signal showsa sound source change structure (change) or a multiple sound sourcechange structure (multiple change) on the basis of the descriptor.

If it is determined in Step S13 that the audio signal of the block showsneither a sound source change structure (change) nor a multiple soundsource change structure (multiple change), the scene change detectingsection 75 outputs a signal representing the result of detection to thereplay control section 77. Upon receiving the signal representing theresult of detection, the replay control section 77 controls the replaysection 71 so as to make it continue the replay operation in the fastreplay mode. Thus, the processing operation of the embodiment returns toStep S11 and the operations of Steps S11 through 13 are repeated on theaudio signal of the next block.

If, on the other hand, it is determined in Step S13 that the audiosignal of the block shows either a sound source change structure or amultiple sound source change structure there, the scene change detectingsection 75 outputs a signal representing the result of detection to thereplay control section 77. Upon receiving the signal representing theresult of detection, the replay control section 77 controls the replaysection 71 so as to make it continue the replay operation in the fastreplay mode. Then, the classifying section 74 carries out in Step S14 aclassifying operation on the audio signal of the next block that isreproduced in the fast replay mode, using the above described techniquesof blocking, characteristic extraction and vector quantization andoutputs a descriptor (tag) showing the result of classification of eachblock.

Then, in Step S15, upon receiving the descriptor showing the result ofclassification of the block obtained in Step S14, the scene changedetecting section 75 checks in Step S16 if the audio signal shows asilence structure (silent) or not on the basis of the descriptor.

If it is determined in Step S 16 that the audio signal of the blockshows a silence structure, the scene change detecting section 75 outputsa signal representing the result of detection to the replay controlsection 77. Upon receiving the signal representing the result ofdetection, the replay control section 77 controls the replay section 71so as to make it continue the replay operation in the fast replay mode.Thus, the processing operation of the embodiment returns to Step S14 andthe operations of Steps S14 through 16 are repeated on the audio signalof the next block.

If, on the other hand, it is determined in Step S16 that the audiosignal of the block does not show a silence structure there, the scenechange detecting section 75 outputs a signal representing the result ofdetection to the replay control section 77. Upon receiving the signalrepresenting the result of detection, the replay control section 77controls the replay section 71 so as to make it stop the replayoperation in the fast replay mode and start a replay operation in thenormal speed mode. Then, the audio signal reproduced by in the normalspeed mode is transmitted to the loudspeaker of the display apparatus(not shown) connected to the embodiment by way of mixing section 72 andterminal 73. As a result, the sound represented by the audio signal thatis reproduced in the normal speed mode is output from the loudspeaker ofthe display apparatus.

Thus, as it is so determined in Step S 16 that the audio signal of theblock does not show any silence structure there, the audio signal of theblock is regarded as that of the sound of a new scene. Then, theembodiment reproduces the audio signal for the start of a new scene inthe normal speed mode. Therefore, the user can recognize if the soundcoming after the scene change and reproduced in the normal speed mode isthe sound he or she wants or not by listening to the sound withoutfeeling any difficulty. Additionally, as it is so determined in Step S16 that the audio signal of the block does not show any silencestructure there, the signal representing the result of detection is alsosent to notification signal generating section 76 from the scene changedetecting section 75. Upon receiving the signal representing the resultof detection, the notification signal generating section 76 generatesand outputs a notification sound signal for notifying the user of thefact that a scene change is detected. The notification sound signal isthen sent to the loudspeaker of the display apparatus by way of themixing section 72 and a notification sound for notifying the detectionof the scene change is output from the loudspeaker so that the user canrecognize the detection of the scene change. The notification signaloutput form the notification signal generating section 76 may be adisplay signal for showing a message on the detection of a scene changeon the display screen of the display apparatus. It may be appreciatedthat, if a display signal is output from the notification signalgenerating section 75 as notification signal, the signal will betransmitted not to the mixing section 72 but to the display section ofthe display apparatus.

As described above, with the second embodiment of signal processing anapparatus according to the invention, points of change (scene changes)of an audio signal can be detected as a result of classifying the audiosignal in a manner as described earlier by referring to the firstembodiment so that the point of switch of topics or that of televisionprograms and hence multimedia data can be retrieved automatically withease. Additionally, with the second embodiment of signal processing anapparatus according to the invention, the user now can listen only tocandidate parts of signals that may show the start of the scene changehe or she is looking for in the normal speed mode and detects the rightone without being forced to pay efforts for tediously listening to allthe sounds stored in the recording medium in the fast replay mode.

Still additionally, when used with a technique of detecting points ofswitch of cuts (e.g., points of switching cameras shooting scenes), thesecond embodiment of signal processing an apparatus according to theinvention can improve the accuracy of detecting scene changes (unitscenes, or cuts, forming a visual entity).

While the first and second embodiments of the invention are describedabove in terms of audio signals, it may be appreciated that the presentinvention can also be applied to video signals and other signals for thepurpose of classifying them, generating descriptors for them andretrieving them.

1-2. (canceled)
 3. A method for classifying signals comprising: dividingan input signal into blocks having a predetermined time length;extracting one or more than one characteristic quantities of a signalattribute from the signal of each block; and classifying the signal ofeach block into a category according to the characteristic quantitiesthereof, wherein said signal of each block is classified into any of thecategories formed on the basis of types structures that signals may haveand do not depend on the types of signal sources, and wherein one ormore than one of the average and variances of the signal power in theblock, the average and variances of the power of a band-pass signal ofthe signal in the block, the average and variances of the spread of thespectrogram of the signal in the block, the average and variances of thepitch frequency of the signal in the block, the average and variances ofthe degree of harmonic structurization of the signal in the block, theaverage and variances of the residue signal of linear predictiveanalysis of the signal in the block and the average and variances of thepitch gain of the residue signal of linear predictive analysis of thesignal in the block are used as said characteristic quantities.
 4. Themethod for classifying signals according to claim 1, wherein said inputsignal is an audio signal; and the categories formed on the basis of thesignal sources for classifying the audio signal of each block includeone or more than one of silence, voice, male voice, female voice, music,vocal music, instrumental music, noise, striking sound, environmentalsound, sound of hustle and bustle, clapping sound and cheering sound andare used for categorical classification based on the sound sources. 5-6.(canceled)
 7. The method for classifying signals according to claim 3,wherein said average of the degree of harmonic structurization is thetemporal average of the ratio of the energy of the sound component ofinteger times of the pitch of the frequency to the energy of all thefrequencies; and said variances of the degree of harmonicstructurization is the temporal standard deviation of the ratio of theenergy of the sound component of integer times of the pitch frequency tothe energy of all the frequencies.
 8. The method for classifying signalsaccording to claim 3, wherein a vector quantization technique is used asa method for the categorical classification. 9-10. (canceled)
 11. Anapparatus for classifying signals comprising: a blocking means fordividing an input signal into blocks having a predetermined time length.a feature extracting means for extracting one or more than onecharacteristic quantities of a signal attribute from the signal of eachblock; and a categorical classifying means for classifying the signal ofeach block into a category according to the characteristic quantitiesthereof, wherein said categorical classifying means classifies saidsignal of each block into any of the categories formed on the basis oftypes of structures that signals may have and do not depend on the typesof signal sources, and wherein said feature extracting means uses one ormore than one of the average and variances of the signal power in theblock, the average and variances of the power of a band-pass signal ofthe signal in the block, the average and variances of the spread of thespectrogram of the signal in the block, the average and variances of thepitch frequency of the signal in the block, the average and variances ofthe degree of harmonic structurization of the signal in the block, theaverage and variances of the residue signal of linear predictiveanalysis of the signal in the block and the average and variances of thepitch gain of the residue signal of linear predictive analysis of thesignal in the block as said characteristic quantities.
 12. The apparatusfor classifying signals according to claim 52, wherein said input signalis an audio signal; and the categories formed on the basis of signalsources for classifying the audio signal of each block include one ormore than one of silence, voice, male voice, female voice, music, vocalmusic, instrumental music, noise, striking sound, environmental sound,sound of hustle and bustle, clapping sound and cheering sound and areused for categorical classification based on the sound sources. 13-14.(canceled)
 15. The apparatus for classifying signals according to claim11, wherein said average of the degree of harmonic structurization isthe temporal average of the ratio of the energy of the sound componentof integer times of the pitch frequency to the energy of all thefrequencies; and said variances of the degree of harmonicstructurization is the temporal standard deviation of the ratio of theenergy of the sound component of integer times of the pitch frequency tothe energy of all the frequencies.
 16. The apparatus for classifyingsignals according to claim 11, wherein said categorical classifyingmeans uses a vector quantization technique as method for the categoricalclassification. 17-18. (canceled)
 19. A method for generatingdescriptors comprising: dividing an input signal into blocks having apredetermined time length; extracting one ore more than onecharacteristic quantities of a signal attribute from the signal of eachblock; classifying the signal of each block into a category according tothe characteristic quantities thereof, wherein said signal of each blockis classified into any of the categories formed on the basis of types ofstructures that signals may have and do not depend on the types ofsignal sources, and generating a descriptor for the signal according tothe category of classification thereof, wherein one or more than one ofthe average and variances of the signal power in the block, the averageand variances of the power of a band-pass signal of the signal in theblock, the average and variances of the spread of the spectrogram of thesignal in the block, the average and variances of the pitch frequency ofthe signal in the block, the average and variances of the degree ofharmonic structurization of the signal in the block, the average andvariances of the residue signal of linear predictive analysis of thesignal in the block and the average and variances of the pitch gain ofthe residue signal of linear predictive analysis of the signal in theblock are used as said characteristic quantities.
 20. The method forgenerating descriptors according to claim 53, wherein said output signalis an audio signal; and the categories formed on the basis of signalsources for classifying the audio signal of each block include one ormore than one of silence, voice, male voice, female voice, music, vocalmusic, instrumental music, noise, striking sound, environmental sound,sound of hustle and bustle, clapping sound and cheering sound and areused for categorical classification based on the sound sources. 21-22.(canceled)
 23. The method for generating descriptors according to claim19, wherein said average of the degree of harmonic structurization isthe temporal average of the ratio of the energy of the sound componentof integer times of the pitch frequency to the energy of all thefrequencies; and said variances of the degree of harmonicstructurization is the temporal standard deviation of the ratio of theenergy of the sound component of integer times of the pitch frequency tothe energy of all the frequencies.
 24. The method for generatingdescriptors according to claim 19, wherein a vector quantizationtechnique is used as method for the categorical classification. 25-26.(canceled)
 27. An apparatus for generating descriptors comprising: ablocking means for dividing an input signal into blocks having apredetermined time length: a feature extracting means for extracting oneor more than one characteristic quantities of a signal attribute fromthe signal of each block; a categorical classifying means forclassifying the signal of each block into a category according to thecharacteristic quantities thereof wherein said categorical classifyingmeans classifies said signal of each block into any of the categoriesformed on the basis of types of structures that signals may have and donot depend on the types of signal sources; and a descriptor generatingmeans for generating a descriptor for the signal according to thecategory of classification thereof, wherein said feature extractingmeans uses one or more than one of the average and variances of thesignal power in the block, the average and variances of the power of aband-pass signal of the signal in the block, the average and variancesof the spread of the spectrogram of the signal in the block, the averageand variances of the pitch frequency of the signal in the block, theaverage and variances of the degree of harmonic structurization of thesignal in the block, the average and variances of the residue signal oflinear predictive analysis of the signal in the block and the averageand variances of the pitch gain of the residue signal of linearpredictive analysis of the signal in the block as said characteristicquantities.
 28. The apparatus for generating descriptors according toclaim 54, wherein said input signal is an audio signal; and thecategories formed on the basis of signal sources for classifying theaudio signal of each block include one or more than one of silence,voice, male voice, female voice, music, vocal music, instrumental music,noise, striking sound, environmental sound, sound of hustle and bustle,clapping sound and cheering sound and are used for categoricalclassification based on the sound sources. 29-30. (canceled)
 31. Theapparatus for generating descriptors according to claim 27, wherein saidaverage of the degree of harmonic structurization is the temporalaverage of the ratio of the energy of the sound component of integertimes of the pitch frequency to the energy of all the frequencies; andsaid variances of the degree of harmonic structurization is the temporalstandard deviation of the ratio of the energy of the sound component ofinteger times of the pitch frequency to the energy of all thefrequencies.
 32. The apparatus for generating descriptors according toclaim 27, wherein said categorical classifying means uses a vectorquantization technique as method for the categorical classification.33-34. (canceled)
 35. A method for retrieving signals comprising:dividing an input signal into blocks having a predetermined time length;extracting one or more than one characteristic quantities of a signalattribute from the signal of each block; classifying the signal of eachblock into a category according to the characteristic quantitiesthereof, wherein said signal of each block is classified into any of thecategories formed on the basis of types of structures that signals mayhave and do not depend on the types of signal sources and retrieving thesignal according to the result of categorical classification or by usinga descriptor generated according to the result of categoricalclassification, wherein one or more than one of the average andvariances of the signal power in the block, the average and variances ofthe power of a band-pass signal of the signal in the block, the averageand variances of the spread of the spectrogram of the signal in theblock, the average and variances of the pitch frequency of the signal inthe block, the average and variances of the degree of harmonicstructurization of the signal in the block, the average and variances ofthe residue signal of linear predictive analysis of the signal in theblock and the average and variances of the pitch gain of the residuesignal of linear predictive analysis of the signal in the block are usedas said characteristic quantities.
 36. The method for retrieving signalsaccording to claim 55, wherein said input signal is an audio signal; thecategories formed on the basis of signal sources for classifying theaudio signal of each block include one or more than one of silence,voice, male voice, female voice, music, vocal music, instrumental music,noise, striking sound, environmental sound, sound of hustle and bustle,clapping sound and cheering sound and are used for categoricalclassification based on the sound sources; and a signal is retrieved byusing the descriptor reflecting or corresponding to the result of saidcategorical classification based on the sound sources. 37-38. (canceled)39. The method for retrieving signals according to claim 35, whereinsaid average of the degree of harmonic structurization is the temporalaverage of the ratio of the energy of the sound component of integertimes of the pitch frequency to the energy of all the frequencies; andsaid variances of the degree of harmonic structurization is the temporalstandard deviation of the ratio of the energy of the sound component ofinteger times of the pitch frequency to the energy of all thefrequencies.
 40. The method for retrieving signals according to claim35, wherein a vector quantization technique is used as method for thecategorical classification.
 41. The method for retrieving signalsaccording to claim 35, wherein points of changes of the signal aredetected by using the descriptor reflecting or corresponding to theresult of said categorical classification. 42-43. (canceled)
 44. Anapparatus for retrieving signals comprising: a blocking means fordividing an input signal into blocks having a predetermined time length;a feature extracting means for extracting one or more than onecharacteristic quantities of a signal attribute from the signal of eachblock; a categorical classifying means for classifying the signal ofeach block into a category according to the characteristic quantitiesthereof, wherein said categorical classifying means classifies saidsignal of each block into any of the categories formed on the basis oftypes of structures that signals may have and do not depend on the typesof signal sources; and a signal retrieving means for retrieving thesignal according to the result of categorical classification or by usinga descriptor generated according to the result of categoricalclassification, wherein said feature extracting means uses one or morethan one of the average and variances of the signal power in the block,the average and variances of the power of a band-pass signal of thesignal in the block, the average and variances of the spread of thespectrogram of the signal in the block, the average and variances of thepitch frequency of the signal in the block, the average and variances ofthe degree of harmonic structurization of the signal in the block, theaverage and variances of the residue signal of linear predictiveanalysis of the signal in the block and the average and variances of thepitch gain of the residue signal of linear predictive analysis of thesignal in the block as said characteristic quantities.
 45. The apparatusfor retrieving signals according to claim 56, wherein said input signalis an audio signal; the categories formed on the basis of signal sourcesfor classifying the audio signal of each block include one or more thanone of silence, voice, male voice, female voice, music, vocal music,instrumental music, noise, striking sound, environmental sound, sound ofhustle and bustle, clapping sound and cheering sound and are used forcategorical classification based on the sound sources; and said signalretrieving means retrieves a signal by using the descriptor reflectingor corresponding to the result of said categorical classification basedon the sound sources. 46-47. (canceled)
 48. The apparatus for retrievingsignals according to claim 44, wherein said average of the degree ofharmonic structurization is the temporal average of the ratio of theenergy of the sound component of integer times of the pitch frequency tothe energy of all the frequencies; and said variances of the degree ofharmonic structurization is the temporal standard deviation of the ratioof the energy of the sound component of integer times of the pitchfrequency to the energy of all the frequencies.
 49. The apparatus forretrieving signals according to claim 44, wherein said categoricalclassifying means uses a vector quantization technique as method for thecategorical classification.
 50. The apparatus for retrieving signalsaccording to claim 44, wherein said signal retrieving means detectspoints of changes of the signal by using the descriptor reflecting orcorresponding to the result of said categorical classification.
 51. Themethod for classifying signals according to claim 3, wherein said signalof each block is classified into any of the categories formed on thebasis of types of signal sources.
 52. The apparatus for classifyingsignals according to claim 11, wherein said categorical classifyingmeans classifies said signal of each block into any of the categoriesformed on the basis of types of signal sources.
 53. The method forgenerating descriptors according to claim 19, wherein said signal ofeach block is classified into any of the categories formed on the basisof types of signal sources.
 54. The apparatus for generating descriptorsaccording to claim 27, wherein said categorical classifying meansclassifies said signal of each block into any of the categories formedon the basis of types of signal sources.
 55. The method for retrievingsignals according to claim 35, wherein said signal of each block isclassified into any of the categories formed on the basis of types ofsignal sources.
 56. The apparatus for retrieving signals according toclaim 44, wherein said categorical classifying means classifies saidsignal of each block into any of the categories formed on the basis oftypes of signal sources.